Evaluating MapReduce for Multi-core and Multiprocessor Systems

As multi-core chips become ubiquitous, it is critical to develop parallel programming models and runtime systems that can harness their computational capabilities. In this paper, we evaluate the suitability of the MapReduce model for multi-core and multi-processor systems. MapReduce was developed by Google to program and manage data-centers with thousands of servers. It allows programmers to write functional-style code that is automatically parallelized and scheduled on a distributed system. We describe Phoenix, an implementation of MapReduce for shared-memory parallel systems that includes a programming API and an efficient runtime system. The Phoenix runtime automatically manages threadgeneration, dynamic task scheduling, data partitioning, and fault-tolerance across the processor nodes. We evaluate Phoenix with a diverse set of benchmarks on both multi-core and symmetric multiprocessor systems. We demonstrate that Phoenix leads to excellent speedups with both systems types. The speedups are robust across a wide range of system and dataset characteristics. We also show that Phoenix can automatically recover from transient and permanent errors on map and reduce tasks. Finally, we compare to parallel versions of the benchmarks written in the lower-level Pthreads API and demonstrate that despite the overheads associated with the MapReduce model, Phoenix leads to competitive performance with significantly simpler code. Overall, we establish that MapReduce provides a promising method for programming and managing parallel applications on multi-core and other shared-memory systems.